-
Notifications
You must be signed in to change notification settings - Fork 1.7k
Overlay: Add overlay annotations to Java & shared libraries #19779
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
base: kaspersv/overlay-annotations-script
Are you sure you want to change the base?
Overlay: Add overlay annotations to Java & shared libraries #19779
Conversation
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Pull Request Overview
This PR adds overlay[local?]
and module;
declarations across Java (and some C#) QL library files to opt them into the new overlay compilation mode, and applies overlay[caller]
to inline public predicates to preserve inlining across overlay boundaries.
- Insert
overlay[local?]
andmodule;
at the top of selected Java/QSharp QLL files following the heuristic in the PR description. - Add
overlay[caller]
beforepragma[inline]
on public, inline-marked predicates in dataflow modules to ensure they remain inlined across overlay frontiers.
This is currently a fairly incomplete heuristic (there are other places that invoke global data flow), but I guess this provides a decent enough starting point and that further manual tweaking is to be expected. |
Agreed. I think we should keep the current heuristic for now as we have tested the results extensively. However, if you have concrete ideas for extending the heuristic, I would be happy to try to incorporate them in the future. |
@hvitved @tausbn Would you mind reviewing the C# and Python parts of this PR (commit 2)? The PR adds overlay annotations for Java, but since we still have a few files that are sync'ed between languages, this also affects a few C# and Python files. Overlay compilation is still disabled for C# and Python and will remain so for the foreseeable future, so the annotations won't have any affect on compilation for C# or Python, beyond additional error checks. The overlay annotations are documented here and I would also be happy to give a quick intro to the overlay annotations as context. |
5f5e0d9
to
ed8b787
Compare
ea40677
to
052023e
Compare
This PR adds overlay annotations for Java libraries and shared libraries to support experimentation with Java overlay analysis. Overlay annotations were added automatically using the
add-overlay-annotations.py
script. The high-level intend is that dataflow should be global and as much as possible below dataflow should be local. To achieve this the script adds top-leveloverlay[local?]
annotations to Java and shared libraries based on a simple heuristic:Test.qll
Query.qll
orConfig.qll
if they containimplements DataFlow::ConfigSig
.For files selected for annotation, the script also adds
overlay[caller]
annotations on all public predicates annotated withpragma[inline]
to ensure that those predicates will still be inlined across the overlay frontier once overlay compilation is enabled. See the internal Incremental CodeQL docs for additional details.I recommend reviewing this PR by reviewing the script that generated the annotations.
Some of the annotated Java library files are also used by Python and C# and the annotations have therefore also been added to the Python and C# variants through
sync-files.py
.This PR does not enable overlay compilation for Java and therefore currently has no effect on the generated DIL, RA or QLX (as witnessed by the uneventful DCA runs). The only current impact of adding the annotations will be that the compiler will check for overlay annotation errors, but in the absence of errors the overlay annotations currently have no effect on compilation. See the internal Incremental CodeQL docs for additional details.
A CI check will be added in a subsequent PR to enforce usage of the
add-overlay-annotations.py
script to automatically add overlay annotations to newly added files.